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RESOLUTION IMAGE OUTPUT 

Inventor 
Joshua I. Pine 

Background of the Invention 

1. Technical Field 

[001] The invention relates generally to a method of and apparatus for increasing the 
effective resolution of a digital imager. 

Background of the Invention 

2. Description of tlie Related Art 

[002] Over the past number of years, digital cameras have begun to replace more 
conventional film camera. Digital cameras employ electronic devices, or "imagers," to 
capture a picture by using electronic components. Originally, the technology of 
electronic image capture employed charged-coupled devices (CCDs) in products such as 
a television camera in 1975 and an observatory telescope in 1979. In 1982, the first 
solid state personal camera using CCDs was introduced. 

[003] A low cost and more commonly available electronic component that is suitable 
for imagers is a complementary metal oxide semiconductor (CMOS) sensor. One 
advantage of CMOS sensors is that, unlike CCD technology, other circuits such as those 
required for error correction, image stabilization, image compression and image 
enhancement may be incorporated in the same chip. In addition, a single chip design 
typically requires less power than a multiple chip design, thus increasing battery life. 
[004] Typically, a picture produced by all but the most expensive digital camera does 
not have as fine a resolution as the more traditional film-based cameras. CMOS imager 
sensors included in digital cameras are fabricated as a two-dimensional array of millions 
of pixels. Until now, the only way to increase the resolution of the pictures the digital 
camera produces is to increase the number of pixels. Of coarse, increasing the number 
of pixels also increases the cost and complexity of the imager. 
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Summary 

[005] Digital image enhancement techniques and apparatus are provided that increase 
the resolution of a digital imager without requiring an increase in the number of pixel 
sensors in the digital imager. Multiple images are captured in succession from the 
digital imager, typically following a time interval corresponding to the length of time it 
takes to capture and store each image. The multiple images are compared m order to 
determine a correlation between the pixels of each image and the pixels of each of the 
other images. Two or more of the multiple images are employed to produce a single 
image of greater resolution than the resolution of any single image alone. The 
technique takes advantage of a change in position of the digital imager between the 
capture of individual images and the resultant differences in the captured images. These 
resultant differences represent additional data that is employed to create a single image 
with a higher resolution than a digital imager is typically capable of producing. In other 
words, a digital imager with two million sensors can produce an image with a resolution 
equivalent to a digital sensor with four million pixels or more. 

[006] The techniques of the disclosed embodiment may be implemented in circuitry 
that is either co-located with an imaging an-ay or located on an additional electronic 
component coupled to the imaging array. An advantage to the technique is that it 
reduces the need for addition pixel sensors to produce a higher resolution image. The 
technique can be implemented either in real time, as the digital images are produced, or 
implemented offline on successive images that have been stored in an electronic storage 
medium. 

[007] Other systems, methods, features and advantages of the invention will be or will 
become apparent to one with skill in the art upon examination of the following figures 
and detailed description. It is intended that all such additional systems, methods, 
features and advantages be included within this description, be within the scope of the 
invention, and be protected by the accompanying claims. 
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Brief Description Of The Figures 

[008] The invention can be better understood with reference to the following figures. 
The components in the figvu-es are not necessarily to scale, emphasis instead being 
placed upon illustrating the principles of the invention. Moreover, in the figures, like 
reference numerals designate corresponding parts throughout the different views. 
[009] Figure 1 illustrates an exemplary imaging array. 

[010] Figure 2 illustrates two successive images from the imaging array of Figure 1 
superimposed upon one another. 

[Oil] Figure 3 illustrates possible positions for pixels of successive images with 
respect to each other. 

[012] Figure 4 illustrates several possible relationships between the corresponding 
pixels of the two successive images of Figure 2. 

[013] Figure 5 is a flowchart of one embodiment of the disclosed digital image 
enhancement techniques. 
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Detailed Description 

[014] Although described with particular reference to a soUd state imager, the image 
enhancement device (lED) of the invention can be implemented in any system in which 
it is desirable to increase the effective resolution of an image without increasing the 
number of hardware components. In addition, although described in conjunction with a 
color imager, the disclosed techniques are equally applicable to a monochrome imager. 
[015] The lED of the invention can be implemented in software, hardware, or a 
combination of hardware and software. Selected portions of the lED are implemented 
in hardware and software. The hardware portion of the mvention can be implemented 
using specialized hardware logic. The software portion can be stored in a memory and 
be executed by a suitable instruction execution system (microprocessor). The hardware 
implementation of the lED can include any or a combination of the following 
technologies, which are well known in the art: a discrete logic circuit(s) having logic 
gates for implementing logic fimctions upon data signals, an application specific 
integrated circuit having appropriate logic gates, a programmable gate array(s) (PGA), a 
field programmable gate array (FPGA), etc. 

[016] Furthermore, the software of the lED, which comprises an ordered hsting of 
executable instructions for implementing logical functions, can be embodied in any 
computer-readable medium for use by or in connection with an instruction execution 
system, apparatus or device, such as a computer-based system, processor-containing 
system or other system that can fetch instructions from the instruction execution system, 
apparatus or device and execute the instructions. 

[017] In the context of this document, a "computer-readable medium" can be any 
means that can contain, store, communicate, propagate or transport the program for use 
by or in connection with the instruction execution system, apparatus or device. The 
computer-readable medium can be, for example but not limited to, an electronic, 
magnetic optical, electromagnetic, infrared or semiconductor system, apparatus or 
device. More specific examples (a nonexhaustive Hst) of the computer readable 
medium would include the following: an electrical connection (electronic) having one 
or more wires, a portable computer diskette (magnetic), a random access memory 
(RAM), a read-only memory (ROM, an erasable programmable read-only memory 
(EPROM or Flash memory), (magnetic), an optical fiber (optical) and a portable 
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compact disc read-only memory or another suitable medium upon which the program is 
printed, as the program can be electronically captured, via for instance optical scanning 
of the paper or other medium, then compiled, interpreted or otherwise processed in a 
suitable manner if necessary, and then stored in a computer system. 
[018] Turning now to the figures. Figure 1 illustrates an imaging array 100 that 
includes thirty-six pixel sensors 11-16, 21-26, 31-36, 41-46, 51-56 and 61-66. 
Typically, an imager array contains millions of pixel sensors, but, for simplicity, the 
illustrated imaging array 100 contains only thirty-six pixel sensors. In the disclosed 
embodiment, each pixel sensor is a charged-coupled devices (CCD) that builds a charge 
when exposed to light of a wavelength of a particular color. In alternative 
embodiments, the techniques may be implemented in other types of electronic devices 
such as a complementary metal oxide semiconductor (CMOS) device and a silicon 
germanium (SiGe) device In this maimer, the imaging array captures an image that is 
then stored in a digital format. In order to capture an image, the pixel sensors 11-16, 
21-26, 31-36, 41-46, 51-56 and 61-66 of the imaging array 100 are exposed to light, the 
pixel values are read, processed according to the techniques of the disclosed 
embodiment, and recorded. An image that is captured at a particular time within the 
pixel sensors 11-16, 21-26, 31-36, 51-56 and 61-66 of the imaging array 101 is referred 
to as a "frame." Each frame captures a digital representation of the scene, or "view," at 
which the digital imager 100 is pointed. 

[019] In this example, the imaging array is configured in a "Bayer" pattern. A Bayer 
pattern consists of alternating rows with alternating colors in each row; i.e., alternate 
rows 10, 30 and 50 are similar, and alternate rows 20, 40 and 60 are similar. Rows 10, 
30 and 50 include alternating blue ("B") and green ("G") pixel sensors. In row 10, 
pixels 11, 13 and 15 are blue; and pixel sensors 12, 14 and 16 are green. Rows 20, 40 
and 60 include alternating green and red ("R") pixel sensors. In row 20, pixel sensors 
21, 23 and 25 are green; and pixel sensors 22, 24 and 26 are red. This alternating colors 
in alternating rows is relevant to the techniques of the disclosed embodiment because 
between successive captured images the imaging array tj^ically moves a slight amount. 
The techniques compare pixel sensors of the same color to determine a correspondence 
between successive images. Because of the alternating pixel sensor colors, the imaging 
array must move either almost no distance or at least a two-pixel distance for pixel 
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sensors of the same color in corresponding images to represent the same or close to the 
same point in a scene captured by the pixel sensors. The techniques take advantage of 
the difference between images; and, if pixels of the same color in corresponding images 
represent the same point, little image resolution is gained. Thus, the two-pixel distance 
between pixel sensors of the same color provide a wider timing window for successive 
images. 

[020] Also included in the imaging array 100 is a read out register 102. The read out 
register 102 includes read out bytes (ROBs) 71-76. In order to record the values of the 
pixel sensors 11-16, 21-26, 31-36, 41-46, 51-56 and 61-66, the pixel values are shifted 
into the read out register 102 and then transmitted sequentially, producing an 
IMAGE OUT signal 104. ROBs 71-76 correspond to columns 81-86 of pixel sensors, 
respectively. During a read of the imaging array, columns 81-86 are shifted into 
corresponding ROBs 71-76 one row art a time. For example, the value of the pixel 
sensor 61 of column 81 is shifted into the ROB 71, the value of pixel sensor 51, also of 
column 81, is shifted into pixel sensor 61, and so on. In other words, once the pixel 
sensors 61-66 of row 60 are shifted into the ROBs 71-76 of the read out register 103, the 
ROBs 71-76 are transmitted sequentially as the IMAGE OUT signal 104. Then, row 
50 is shifted into row 60, row 40 is shifted into row 50 and so on. This process is 
continued until the values of all the pixel sensors 11-16, 21-26, 31-36, 41-46, 51-56 and 
61-66, or the frame, have been transmitted as the IMAGE_OUT signal 104. 
[021] The imaging array 100 transmits the IMAGE_OUT signal 104 to an image 
enhancement device 108. The lED 108 includes a memory 106 for storing data 
transmitted on the IMAGE_OUT signal 104 and for storing code to implement the 
techniques of the disclosed embodiment. The memory 106 may be implemented as 
registers, random access memory (RAM), first-in, first-out (FIFO) memory, or some 
other well known memory system. The specific type of memory 106 is not critical to 
the spirit of the invention. A CONTROL signal 1 12 is transmitted from the lED 108 to 
the imaging array 100. The ftmction of the CONTROL signal 112 is explained in more 
detail below in conjunction with Figure 5. Also transmitted from the lED 108 is an 
ENHANCED_IMAGE_OUT ("EJ O") signal 110. The E_I_0 signal 110 represents 
an enhanced image produced according to the techniques of the disclosed embodiment. 
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[022] Figure 2 illustrates two successive images, an image_X 10 and an image_Y 20, 
both of which include pixel values read from the imaging array 100 (Fig. 1). Image_X 
10 and image_Y 20 are superimposed on each other. Image_X 10 includes pixel values 
Bx 111, Gx 112, Bx 113, Gx 114, Bx 115, Gx 116, Gx 121, Rx 122, Gx 123, Rx 124, 
Gx 125, Rx 126, Bx 131, Gx 132, Bx 133, Gx 134, Bx 135, Gx 136, Gx 141, Rx 142, 
Gx 143, Rx 144, Gx 145, Rx 146, Bx 151, Gx 152, Bx 153, Gx 154, Bx 155, Gx 156, 
Gx 161, Rx 162, Gx 163, Rx 164, Gx 165 and Rx 166. Image_Y 20 includes pixel 
values By 21 1, Gy 212, By 213, Gy 214, By 215, Gy 216, Gy 221, Ry 222, Gy 223, Ry 
224, Gy 225, Ry 226, By 231, Gy 232, By 233, Gy 234, By 235, Gy 236, Gy 241, Ry 
242, Gy 243, Ry 244, Gy 245, Ry 246, By 251, Gy 252, By 253, Gy 254, By 255, Gy 
256, Gy 261, Ry 262, Gy 263, Ry 264, Gy 265 and Ry 266. 

[023] The pixel values 111-116, 121-126, 131-136, 141-146, 151-156, 161-166, 211- 
216, 221-226, 231-236, 241-246, 251-256 and 261-266 of image_X 10 and image_Y 20 
represent pixel values captured by successive readings of the pixel sensors 11-16, 21-26, 
31-36, 41-46, 51-56 and 61-66 (Fig. 1). Each of the pixel sensors 11-16, 21-26, 31-36, 
41-46, 51-56 and 61-66 captures a hght value corresponding to a spot in a scene, or 
"view," 200 to which the imaging array 100 is pointed. A complete image such as one 
represented by the pixel values 111-116, 121-126, 131-136, 141-146, 151-156, 161-166 
typically takes 30 milliseconds (ms) to be transferred from the imaging array 100, 
through the RGBs 71-76, transmitted out the IMAGE_OUT signal 104 (Fig. 1) and 
stored in the memory 106. In this 30 ms time interval, the position of the imaging array 
100 may sift a small distance with the result that the image_X 10 represents a shghtly 
different frame than the image Y 20. The techniques of the disclosed embodiment take 
advantage of this small difference between image_X 10 and image_Y 20 to produce a 
single image with a higher resolution than either image_X 10 or image_Y 20 alone. 
The position of image_X 10 with respect to image Y 20 depends upon the amount the 
imaging array 100 has moved in the time interval between the two images 10 and 20. 
The time interval between the capture of image X 10 and image_Y 20 may be between 
the minimum time necessary to read out an image from the imaging array 100, or 30 ms 
in this example, to a djoiamically-determined of fixed interval of time, such as 100 ms. 
[024] Figure 3 illusfrates a four-pixel sensor portion of the imaging array 100, 
specifically pixel sensors B 1 1, B 13, B 31 and B 33. In this example, the pixel sensors 
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B 1 1, B 13, B 31 and B 33 are assumed to be in position to capture the pixel values Bx 
lll,Bx 113, Bx 131 and Bx 133. After the imaging array 100 shifts position between 
the capture of image X 10 and image Y 20, the pixel B 1 1 captures pixel value By 21 1 
of image_Y 20. After the imaging array 100 shifts position between the capture of 
5 image_Y 20 and an image_Z (not shown), the pixel B 1 1 captures a pixel value Bz 31 1 

of image Z. An area X 302 is the space between the pixel sensors B11,B13,B31 
and B 33 and represents some possible pixel value capture locations of the pixel sensor 
B 11, assuming that the imaging array 100 has shifted in the direction of area_X 302 
and not beyond. 

10 [025] Included within area X 10 are four areas of equal size, an area_Xl 304, an 

~ I area_X2 306, and area_X3 308 and an area_X4 310, adjacent to pixel sensor B 13, pixel 

sensor B 33, pixel sensor B 21 and pixel sensor B 11 respectively. When added 
together, area_Xl 304, area_X2 306, area_X3 308 and area_X4 310 represent one half 
(1/2) of the area of area_X 302. An area_XO 312 is the area within area_X 302 that 

15 does not include area_ XI 304, area_X2 306, area_X3 308 or area_X4 310. Area_XO 

312 also represents one half (1/2) of the area of area X 302. If the pixel value By 21 1 
represents a value of light measured somewhere within the area_X 302, then pixel value 
By 211 has an approximately fifty percent (50%) probability of representing a pixel 
sensor reading in area_XO 312 and an approximately fifty percent (50%) probability of 

20 representing a pixel sensor reading in one of area_Xl 304, area_X2 306, area_X3 308 

and area_X4 310. Although the resolution of a resultant image produced according to 
the disclosed techniques from image_X 10 and image_Y 20 is better than either 
image_X 10 or image Y 20 regardless of where within area_X 302 that pixel sensor B 
1 1 1 is located, the resultant image is typically more desirable the greater the separation 

25 between the pixel sensors when image_X 10 is captured and when image Y 20 is 

captured. In other words, it is desu-able for pixel value By 211 to fall within area_X0 
312 rather than area Xl 304, area_X2 306, area_X3 308 or area_X4 310. An area_X5 
314 represents the portion of area XO closest to the pixel sensor B 33. If a pixel sensor 
reading falls within area_X 302 but outside of both area_X2 306 and area_X5 314, then 

30 the pixel value capture position is closer to one of the other pixel sensors B 11, B 13 or 

B31. 
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[026] If the third successive image, image_Z, is captured by the imaging array 100, 
then a third point, the pixel value Bz 31 1, is captured. The pixel value Bz 311, like the 
pixel value By 211, has a fifty percent (50%) probability of falling within area_XO 312. 
Assuming that the positions of the pixel values By 21 land the pixel value Bz 31 1 with 
5 respect to the pixel sensors B 11, B 13, B 31 and B 33 are independent of each other, 
then the probability that at least one of pixel value B 211 and pixel value B 311 falls 
within the area_XO is approximately seventy-five percent (75%), based upon the 
following probability equation: probabihty p ^ 1 - [.5 * .5]. 

[027] It should be noted that although there is approximately a seventy-five percent 
10 (75%) probability that one of pixel values 211 and 311 represents a light reading within 
the optimal area for image enhancement with respect to image_X 10, or area_XO 312, 
there is also a fifty percent (50%) probability that image_Y 20 and image Z are at the 
optimal spacing with respect to each other regardless of their positions with respect to 
image X 10. Thus, the probability of any two images of the images image_X 10, 
f 5 image_Y 20 and image_Z are optimally spaced is approximately eighty-seven percent 

; (87%). Of course, the probability that any two of a number of images are spaced 

f]l optimally with respect to each other approaches one hundred percent (100%) as the 

number of images increases. 

[028] Figure 4 illustrates several possible correlations between the pixel values in 
20 image_X 10 and the pixel values in image_Y 20. Although it is possible for the pixels 

of the imaging array 100 to move farther between images than the two-pixel distance 
assumed for illustrative purposes in Figure 3, the techniques of the disclosed 
embodiment still apply. A greater than two-pixel movement of the imaging array 100 
merely complicates a correlation step 508 (see Fig. 5) without substantially changing 
25 other portions of the disclosed embodiment. 

[029] In this example, Image_Y 20 is shifted with respect to image_X 10 until a 
correct correlation is found. A shift 402 moves image_Y 20 so that pixel value By 211 
is compared with pixel value Bx 111, pixel value Gy 212 is compared with pixel value 
Gx 1 12, pixel value By 213 is compared with pixel value Bx 1 13, and so on. A shift 
30 406 moves image_Y 20 with respect to image_X 10 so that pixel By 211 is compared 

with pixel value Bx 113, pixel value Gy 221 is compared with pixel value Gx 123, pixel 
value By 231 is compared with pixel value Bx 133, and so on. A shift 404 moves 



044368.0376 AUSTIN 244353 v2 



9 



us Express Mail No. EL8 1 3906002US 
Filing Date : October 23, 2001 



PATENT 
Atty. Dkt.: 00CXT0434I 



image Y 20 with respect to image X 10 so that pixel By 211 is compared with pixel 
value Bx 131, pixel value Gy 212 is compared with pixel value Gx 132, pixel value By 
213 is compared with pixel value Bx 133, and so on. 

[030] Shifts 402, 404 and 406 are used as examples only. If the imaging array 100 
5 moves a great enough distance between image X 10 and image_Y 20, then the best 

correlation between image_X 10 and image Y 20 may be to shift image Y 20 several 
pixels distance in one direction or another. It should be noted that the term "shift" 
means to correspond pixels in a particular position; there is no actual shifting of pixels. 
After correlating the pixels values of image X 10 and image_Y 20 in a particular 
ICt position, values of corresponding pixels may be multiplied together and the values of all 
. I the products added together. The position that produces the highest sum of squares is 
then designated the position in which the pixels of image_X 10 and image_Y 20 
= . s correspond to each other. One with skill in the imaging arts knows various methods of 

-'^ correlating images and any method may be employed in the disclosed techniques. 

%5 [031] Figure 5 is a flowchart of a process 500 implemented by the lED 108. Process 

500 begins in a Begin Image Enhancement step 502 and then proceeds directly to a 
Capture Image step 504. In step 504 the pixel sensors 11-16, 21-26, 31-36, 41-46, 51- 
56 and 61-66 of the imaging array 100 (Fig. 1) are exposed to a hght source and become 
charged based upon the intensity of the light source. As explained above in conjunction 
20 with Figure 1 , the values of the pixel sensors 11-16,21 -26, 3 1 -36, 4 1 -46, 5 1 -56 and 6 1 - 

66 are shifted into the ROBs 71-76 of the read-out register 102 and then transmitted as 
the IMAGE_OUT signal 104. For the purposes of this example, the first image is 
image_X 10 (Fig. 2). In a CMOS imaging array, this process typically takes 
approximately 30 ms. 

25 [032] Once the image is captured in step 502, process 500 proceeds to an Enough 

Images step 506 where the process 500 determines whether enough images have been 
captured in step 504. In this example, only one image, image_X 10, has been captured 
so process 500 proceeds to step 504 where another image, image_Y 20, is captured. 
Process 500 then proceeds to step 506 where again process 500 determines whether 

30 enough images have been captured. The determination of whether enough images have 

been captured can be based upon a fixed number of images, such as two or three, or can 
be based upon the properties of the images themselves. In one embodiment, the 
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imaging array 100 automatically produces exactly two images in succession and 
transmits them by means of the IMAGE_OUT signal 104 for processing by the lED 
108. In another embodiment, the lED 108 determines whether another image is 
necessary and issues a CONTROL signal 1 12 to the imaging array. Upon receipt of the 
5 CONTROL signal 1 12, the image array 100 captures another image for transmission on 

the IMAGE OUT 104 signal. For example, the lED 108 can determine in real time 
whether any two of multiple images have the necessary degree of separation, as 
explained above in conjunction with Figures 3 and 4, to enable the lED 108 to produce 
a suitable image. It should be noted that once two images have been correlated, the 

l& degree of separation is inversely proportional to the similarity of the pixel values. In 

other words, if the pixel values of two correlated images are very similar, then there is a 
good probability that the like-color pixel sensors in the imaging array 100 captured 
\ successive values from close to the same spot in the view 200 (Fig. 2). A greater 

separation between images is desirable because more information is then captured for 

1 5 the disclosed enhancement techniques. 

[033] If none of the multiple images have the necessary degree of separation from at 
least one of the other multiple images, then process 500 signals the imaging array 100 
via the control signal 112 and control returns to step 504 where another image is 
captured by the imaging array 100. In another embodiment, the imaging array 100 

20 automatically captures a fixed number of images and transmits those images to the lED 

108 without employing the CONTROL signal 1 12. 

[034] If process 500 determines in step 506 that enough images have been transmitted 
by the imaging array 100, then control proceeds to a correlate images step 508. As 
explained above, in an alternative embodiment, step 508 may be positioned between 

25 step 504 and step 506 in order to assist the determination of whether enough images 

have been captured in step 504. The Correlate Images step 508 functions was explained 
above in conjrmction with Figure 5. The image shifts 402, 404 and 406 represent three 
possible correlations between two images such as image X 10 and image Y 20. It 
should be noted that other shifts are possible if image_X 10 and image_Y 20 are 

30 oriented in different positions with respect to each other. For example, if the imaging 

array 100 has moved more that a two pixel distance between the capture of image_X 10 



044368.0376 AUSTIN 244353 v2 



11 



us Express Mail No. EL813906002US 
Filing Date : October 23, 2001 



PATENT 
ATTY. Dkt.: 00CXT0434I 



and the capture of image_Y 20, pixel value By 213 may, for example, correspond to 
pixel value Bx 1 1 1, with all the other pixels shifted accordingly. 

[035] Following the correlation of the images in step 508, the process 500 proceeds to 

an Interpolate Points step 510 where the images correlated in step 508 are merged by 
5 combining the pixel values from the images into a single image. In this manner, a 

imaging array 100 with two milHon image sensors such as pixel sensors 11-16, 21-26, 

31-36, 41-46, 51-56 and 61-66 can approach the resolution of an imaging array with 

four million or more pixels without increasing the number of pixel sensors themselves. 

[036] After step 510, the process 500 proceeds to a Output Enhanced Image step 512 
10 where the single image produced in step 510 is transmitted on the E_I_0 signal 112 

(Fig. 1). Following the transmission of the single image, the process 500 proceeds to a 
-",| End Image Enhancement step 5 14 where processing is complete. 

: [037] In an alternative embodiment of the claimed subject matter, multiple images 

may be taken and stored before any of the processing steps are initiated. In other words, 
f:^ the disclosed technique may be implemented off-line. In addition, the claimed subject 

7' matter is applicable to the capture and processing of monochrome images. While 

various embodiments of the application have been described, it will be apparent to those 
r=r of ordinary skill in the art that many more embodiments and implementations are 

possible that are within the scope of this invention. Accordingly, the invention is not to 
20 be restricted except in light of the attached claims and their equivalents. 
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